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In February of 1988, four Ontario school boards in 
the metropolitan Toronto area, the Fc culty of Education at the 
University of Toronto, and the Ontario Institute for studies in 
Education formed the Learning consortium. The consortium constitutes 
a partnership to initiate innovative programs in field-based teacher 
education, with an emphasis on improving the quality of education in 
Canadian schools. Over a three-year period, each partner will 
contribute people, time, and money to an expanding and increasingly 
complex set of activities. This summary formative evaluation briefly 
reviews, from the evaluator's perspective, the initial activities of 
the Consortium, conceptualizes and examin* the meaning of formative 
evaluation with reference to the experiences of the firct year, and 
argues the importance of reflexivity to the formative evaluation 
process. The basic position forwarded is that both Consortium 
participants and the evaluator act as formative evaluators. As a 
whole, the consortium provides a forum for considering the conceptual 
and empirical possibilities of formative evaluation. During its first 
year, the planning group consisting of representatives of each of the 
six organizations and the full-time consortium coordinator met 
monthly to develop, manage, and review the program. An approach of 
methodological and programmatic reflexivity was established for 
participants and the evaluator in development of the formative 
evaluation and of the improvement program. (TJH) 
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In February, 1988, four Ontario school boards in the metropolitan Toronto 
area, the Faculty of Education at the University of Toronto (FEUT), and the 
Ontario Institute for Studies in Education (OISE) formed the Learning Consortium, 
a partnership to initiate innovative programs in field-based teacher education. 
For a three year period, each partner contributes people, time, and money to an 
expanding and increasingly complex set of Consortium activities. I air involved as 
a formative evaluator responrible for documenting, critiquing, and facilitating 
the development of the Consortium. 

In the follov/ing sections, I briefly review the initial activities of the 
Consorti um, conceptual ize and then examine the meaning of formative evaluation in 
the experiences of the first year, and argue the importance of reflexivity to the 
formative evaluation process. The basic position of the paper is that both the 
Consortium participants and myself are formative evaluators, situated differently 
v/ithin the program and governed by different emphases and expectations, yet faced 
with similar challenges and endeavours. We snare, 1:0 different degrees, the self- 
conscious experience of building understanding, supporting our mutual development, 
and facilitating implementation v/ithin the program. 
Learning Cciisortixmi: The First Year 

The aim of the Consortium is to improve the quality of education for schools. 
Generally t.ie six partners work together to initiate and implement programs for 
preservice, beginning, and experienced educators, to evaluate these programs and 
generate insights into teacher development, and to disseminate new knowledge and 
practices. In each activity, the Consortium embodies and encourages inquiry, 
reflection, and collaboration (Watson, 1988). 

In the first year, the planning group consisting of representatives from 
each of the six organizations and the full-time Consortium coordinator met monthly 
to develop, manage, and review the program. Some of tne major initiatives include 
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the following: 

Sunimer Institute: two v^eek intensive training in cooperative learning, coaching, 
and management of change. 100 teachers, resource people, and administra- 
tors from the four school boards and FEUT attended. 
Follow-up and Support: broad range of workshops and school visitations to assist 
the Summer Institute participants in the implementation of cooperative 
learning. Tv/o consultants seconded to the Consortium and FEUT respond 
to most of the requests for support. 
Cadre of Traine-^'s: ten days of training dispersed over a six month period to 
aevelop a group o-^ specialists in classroom management, instructional 
skills and strategies, and cooperative learning. Forty teachers and 
consultants from the six Consortium partners are involved. 
Other important but less formal practices are the netv;orking, reflective, and 
monitoring strategies of the coordinator, the bridging and implementation actions 
undertaken by the planning group members within the unique culture of their respec- 
tive organizations, and the emerging linking and capacity-building initiatives by 
Summer Institute participants in their schools. The v;eb and v^ave of Consortium 
activities encompasses and influences many levels within and among the six organi- 
zations. 

I became a faculty member at FEUT six months after t!ie official formation of 
the Consortium. Though the first major initiative, the Summer Institute had not 
occurred, I, as the formative evaluator, had to move in a hurry to catch up with a 
program and Consortium structure that were already changing organically and proceeding 
in many directions. To date, my major evaluation experiences include the following: 

Development, negotiation, and approval of the formative evaluation proposal 

with the planning group. 
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. Participant observation at the two week Summer Institute. 
. Documentation (feedback sheets, semi-structured surveys) of the Summer 
Institute participants' views during, immediately following, and three 
months after the Summer Institute. 
. Informal conversations and open-encled interviews with the coordinator 

and the planning group members. 
. Observation at and increasing involvement in the planning group meetings. 
. Ongoing case study of Summer Institute participants at one school in each 

of the four school boards. 
. Record of relevant documents (memos, reports, agendas, etc.) generated by 

the Consortium, school boards, schools and individuals. 
. Coordination of other inquiries: observation of the cadre of trainers* program 
by the coordinator; log of the activities of the seconded consultants by an 
OISE research assistant; examination of teacher thinking with tv;o Summer 
Institute participants by an OISE graduate student; and case study of teachers 
working with teachers in one of the Consortium schools by an independent researcher 
These experiences provide a foundation for the description and judgment of what 
actually happens, the critique of the program'? fundamental assumptions and values, 
and the examination of the intended, and reali^jd effects of the various Consortium 
activities. The information from formative evaluation builds a database of case 
studies, perspectives, and arguments v/hich enable the Consortium participants to 
engage continuously in critical and constructive program improvement. 
Meaning of Formative Evaluation 

Surprisingly few evaluators have elaborated or proposed alternative constructions 
to Scriver's (1967) original differentiation between formative and summative evaluation 
McClintock (1984) outlines a conceptual framework for the design of formative 
evaluation. In a paper entitled. How can evaluation become formative?^ 
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Fiddy and Stronach (1987) develop an educational theory of the relations between 
evaluations and projects. Herman, Morris and Fitz-Gibbon (1987) elucidate proce- 
dural distinctions between formative and summative evaluation. Numerous formative 
references are also embedded in reports of responsive evaluation (Stake, 1975), 
illuminative evaluation (Parlett, 1981), contextual appraisal (Sirotnik, 1984), 
naturalistic evaluation (Guba, 1987), and action research (Carr & Kemmis, 1986). 
Despite these contributions, most formative evaluations use the language of program 
improvement without conceptualizing the implications of what this means in practice. 

in formative evaluation, improvement occurs through the interrelated pursuit 
of understanding, education, and action. These three aspects of formative evaluation 
are moments in an ongoing, holistic process. Each moment is conspicuous for its own 
set of activities and priorities but ultimately derives meaning from the intercon- 
nection with the other two moments. Both the formative evaluator and the program 
participants are engaged in this improvement process; consequently, both are 
formative evaluators. In essence, formative evaluation is the story of how the 
designated formative evaluator and the program participants forge a partnership in 
the various moments for improvement. 

In the following sub-sections, I elaborate the meaning of each moment, discuss 
some early experiences, and examine the exten^ to which I can or should have similar 
emphases, ^merging from this review is the importance of reflexivity in formative 
evaluation discussed in the final section of the pages. 

Utxiavstandinff 

Improvement evolves from an indepth understanding of how the program is planned, 
supported, experienced, and adapted. Much of the work of formative evaluators has 
traditionally focused on documenting the daily realities of development and imple- 
mentation with a view to generating more effective ways of using and extending the 
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program (Harris, Bell & Carter, 1981; Guba & Lincoln, 1981-, Patton, 1986). This 
requires a closeness to the program, an intimacy tl^at is more defensibly developed 
within a qualitative paradigm. Patton (1987) comments: 

A prim..r'y interest of qualitative-naturalistic evaluators is 
describing and understanding these dynamic program processes 
and their holistic effects on participants so as to provide 
information for program improvement (formative evaluation). 
The qualitative-naturalistic-formative approach is thus 
especially appropriate for programs that are developing, 
innovative, or changing, where the focus is on program 
improvement, facilitating more effective implementation, 
and exploring a variety of effects on participants. This 
can be particularly important early in the life of a program 
or at major points of transition, (pp. 18-19) 
The formative evaluator and program participants begin this searcn for under- 
standing quite a distance apart. The formative evaluator is an uninformed outsider 
trying to gain insights from the understanding which insiders, the program participants, 
already have. Yet there is a movement towards each other, with the formative evaluator 
working "like" a program participant and the program participant working "like" a 
formative evaluator. They rarely meet and share equally the responsibility for 
understanding. Inevitably the formative evaluator remains as the primary researcher 
charged with understanding the p^ ^gram in all its micro and macro complexities. But 
there is a connection where both the formative evaluator and the program participants 
share the following experiences: 

Negotiate what, how, and why information is gathered. 

Adopt the qualitative paradigm to framethe methodology and interpretation of the 
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collected information. 

Build a rich description of how the program unfolds by varying the focus 
(organization, events, resources, support strategies, issues, context, etc.) 
and the unit (individual, classroom, school, school board, etc.) of observation 
and analysis . 

Develop and explore the range of insider's views, comparing the multiple perspec 
tives people have about the program and their positions within the program. 
Become participants in and observers of the program and its formative evaluation 
As these experiences grow, the formative evaluator and the program participants 

construct a mutual understanding of and for improvement. 

Is a partnership in understanding developing in the formative evaluation of the 

Consortium? The Consortium participants realize that they have the understanding 

but assume it is my job to access, record, and assess it. Some participants 

commented: 

YouVe the expert, you decide what should be done. 

My job is to teach, not to research. 

When are you going to measure the outcomes? 

Give us some hard data so we can prove to our trustees that the Consortium 
is successful . 

For these educators, my responsibility is separate from theirs and is method- 
ologically dependent on instruments and other measurement procedures. When the 
participants persist in role-bound perceptions of my involvement and in traditional, 
quantitative views of inquiry, then a sense of partnership is minimal. 

Two developments suggest the emergence of or the potential for partnerships. 
I have frequent and informal conversations with the Consortium coordinator. I 
have regular interactions with the planning group members at their monthly meetings 
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and by telephone. We are developing a working knowledge of each other's situations 
and perspectives. As I participate more, th2y observe more. They structure what 
they know and seek further information to extend their understanding. The line 
between our roles shows signs of blurring as we see the value of working as partners. 

The second development is occurring independent of my actions. Some teachers 
are documenting their cooperative learning experiences in journals or by videotapes. 
In two school boards, the planning group member has sought feedback through informal 
surveys. Other Summer Institute participants are keeping records of their classroom 
or coaching observations. Consortium participants are reflecting on and systematically 
examining their practices. With these educators, I can discuss the Consortium as 
co-investigators and thus begin the journey to a partnership. 

The responsibility of rigorously documenting and reporting the intensive and 
extensive program of the Consortium will remain primarily with me as the participants 
get on with the task of implementing multiple innovations. There is neither an 
exclusivity nor a detachment in this division of labour. We are both trying to 
make sense of the program from our respective vantage points as participants and 
formative evaluator. In this sense-making venture, we raise our consciousness of 
and consequently, build on each other's understanding - a process where understanding 
and education connect. 
Education 

As moments in formative evaluation, understanding and education are inextricably 
tied to one another. The pursuit of understanding is educative; the experience of 
education stimulates understanding. What distinguishes the education moment is the 
emphasis on mutual development in the formative evaluator - program participant 
relationship. Improvement then is linked to the extent to which the formative 
evaluator and the program participants learn from each other. 
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The methodological literature in research and evaluation is replete with 
approaches to guide how the formative evaluator learns about the program and its 
participants (Stake, 1977), Program participants often experience a more restrictive 
mode, learning didactically in the transmission of findings from the formative 
evaluator. This creates a limited pedagogical relationship with little reflection, 
deliberation, or transformation in the ongoing exchanges between the formative 
evaluator and program participants, Fiddy and Stronach (1987) pose a more facilitative 
approach: 

The facilitative approach, soon led to a theory of readiness — 
a notion that the constructive and critical formative evaluation 
is not really possible until participants are ready to accept 
both the process and the evaluator. That 'readiness' is based 
on relationships as well as knowledge. It is governed by a 
pedagogy. Its basic question is not *what is true?', but 'what 
are the educative possibilities in a given situation?'. The 
relative success of the exercise depended to a large extent on 
the alternative strategies available to the evaluator; on his 
ability and desire to discuss its approach; and on the consequent 
shifts in the perception of the evaluation process and of the 
development process by participants, (p. 25) 
Embedded in the facilitative approach is the recognition of expertise that 

both the formative evaluator and the program participants bring to the program 

and its evaluation. Within this reciprocity, they are equally committed to learning 

from the other and their continous interaction. 

As the formative evaluator moves to participation and the program participants 

move to formative evaluation in the pursuit of understanding, they generate insights 
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into each other's world. They begin to explore hov/ each could play a meaningful 
role in the situation of the other. The relationship becomes educative v/hen they 
jointly engage in the following activities: 
Establish an ethos of mutual development. 

Reflect on what, how, and why program and evaluation practices change. 
Encourage a constructive and critical discourse about the program, the 
evaluation, and the link between the tv/o. 

Through dialogue, elaborate the basis for distinct and common interpretations 
of particular problems, operations, and priorities. 

Provide and seek feedback to clarify the meaning of present realities and to 
examine what could and should happen in the future. 

Prolonged collaboration within the action of the program encourages the educative 
partnership in formative evaluation. 

To what extent are my relationships with Consortium participants educative? 
The Consortium participants understand that I learn about the program from what they 
say and do. They expect to learn from spontaneous feedback, formal and informal 
recommendations, and later reports. Some participants quipped: 

Am I using cooperative learning correctly? 

What should I do next? 

Tm not sure if that's what you v/ant in your research. 

You want to know what you can do for me? I don't know. What can you 

do? Give me a few hints. 

You're not going to tell us anything, are you? I'll probably read it 
first in the newspaper. 

These educators demonstrate the p^isive way they see their role within evaluation. 
They feel like sources of data, providing some insights for me .but getting little in 
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return. If they do learn from me, it will be in the form of a quick tip, a 
confirming reaction, or a retrospective 'aha\ The possibilities of an educative 
partnership in this one-way communication are limited. 

Again, in the more extended relationships with the coordinator, the planning 
group members, and some educators in the four schools in which I am conducting 
case studies, there are indications of educative partnerships. I am beginning to 
feel like an '^evaluator in residence" (Bickel, 1984) discussing issues, serving 
as a sounding board for alternative positions and strategies, provoking reaction 
through the expression of dissenting views, and reflecting out loud about the 
dilemmas and contradictions in the promises and practices of the program. My need 
to get close, to cultivate genuine deliberations, to relate as a colleague, to 
open my views for scrutiny, and to affect and be affected by the interaction are 
important conditions in my working relationship with the participants. We are 
teaching each other how to learn from our relationship — the beginnings of an 
educative partnership. 

To be educative is more than an exchange of information. It is more than 
making the other knowledgable about your. perspective on the program. It is a 
dynamic, social process where we probe and negotiate distinct and common inter- 
pretations, interrelate our program and evaluation experiences, and are equally 
committed to each other's learning. Inevitably this pedagogical partnership is 
embedded in the actions of the program itself. 
Action 

Action, when combined with the other moments of education and understanding, 
creates a dialectical tension among the three moments in formative evaluation. 
Program participants try to understand their efforts to plan, use, or adapt program 
practices. They learn from within, through reflecting on, and by interacting about 
these actions. Program actions provide both context and focus to the search for 
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understanding and the educational experience of mutual development. Improvement 
in the action moment refers to the ways in which the formative evaluator and program 
participants facilitate the implementation of the program. 

The literature on facilitating implementation is split, with one set of references 
for program participants (Full an, 1982) and another set of references for evaluators 
(King, 1988; Patton, 1988). In the evaluation literature, the emphasis is on 
utilization. The formative evaluator needs to insinuate himself or herself into the 
program, to get the ear of those who make decisions about program policies and 
interventions, and to respond to the needs of the participants — to make evaluation 
activities count in the daily realities of the program (Bryk & Raudenbush, 1983; 
McCormick & James, 1983). McClintock (1984) describe the implementation role of 
the formative evaluator as a "methodological fool". He states: 

When reducing uncertainty about program performance, the 
evaluator functions (in the traditional manner) as a technician 
or a methodologist. When uncertainty about program structure is 
increased, the evaluator functions like the medieval fool, a char- 
acter who was permitted to make humorous, bizarre, or dangerous 
suggestions that stimulated creativity and catharsis in others. 
To improve effectiveness, there must be information about program 
performance, on the one hand, and alternative strategies and 
structures for program delivery, on the other. Combining these 
kinds of information will maximize the program's potential for 
effectiveness and adaptability in changing environments. Thus, it 
may be preferable to have formative evaluators who are methodological 
"fools" rather than foolish methodologists. (p. 209) 
The above portrayal is formative evaluation "for" improvement. Formative 
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« 

evaluation "as" improvement (Holly & Hopkins, 1988) is a direct implementation 

strategy. Werner (1982) elaborates: 

Continuous evaluation planning and activity is an implementation 
strategy as it identifies emerging strengths, potential difficulties, 
and concerns as they arise at the school. Program impl imentation 
requires that evaluation has a strong formative role. It makes 
participants sensitive to situational factors and to adjust their 
planning accordingly; it allows them to devise implementation 
activities that i'-"-" i build upon the strengths that people have; 
it supplies participants with information of their program, and 
enhances communication; it clarifies the innovation in terms of 
the local situation and participants' beliefs about what is 
important; and it encourages explicit reflection upon the "process 
of putting something new into practice". Through discussion around 
evaluation information, people share beliefs and values, negotiate 
discrepancies and conflicts, and gain reflective distance and under- 
standing of their ov/n situation. Such a process of evaluation is 
change, (p. 2) 

This view establishes a bond between the formative evaluator and the program 
participants as partners in the change process. In particular, they enact the 
following strategies: 

Integrate program development and formative evaluation. 

Work at more than one level of program implementation. 

Raise consciousness about particular approaches, structures, or ideas 

and their consequences for practice. 

Examine alternative change processes, supporting and at times, initiating 
different implementation experiments or interventions. 

er|c ^ ^ ' 
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Determine ways to define, request, ?.nd respond to areas of assistance, 
guidance, or re-direction. 

Though both are change agents, it is the program participants who assume the 
major responsibility in facilitating implementation. The degree to which the 
formative evaluator reaches a similar level of responsibility depends on the nature 
of his or her participation in the program. 

How do my interactions with Consortium participants facilitate implementation? 
Most Consortium participants view my role summatively. To them, my agenda is to 
measure outcomes, confirm successes, or find deficiencies. Some participants 
observed: 

It's really working. 

Tm sorry, I haven't had many chances to try it out in my 
classroom. 

This is the best professional development program Tve ever had. 
How can you say that? That's not what's happening in my school 
board at all . 

I thought you've supposed to evaluate cooperative learning not 
hel p me do it. 

I am the person who wants to know results or needs an explanation about their 
performance. If I express a viewpoint, it implies or states a finding with these 
educators. I am a commentator on implementation, not a partner in action. 

Some joint endeavours are either in negotiation or in their early stages. The 
initial silence to the questions: "How can we work together?" or "What can I do for 
you during the formative evaluation?" now has some answers. The planning group 
members, previously content for me to observe and record at their monthly meetings, 
now note when I have been quiet too long. They invite my reactions, debate my 
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positions, and suggest ways to involve me further. In the schools, I am working with 
a principal - vice principal team to develop alternative school-abased implementation 
strategies and a teacher to study the social and educational situations of a small 
group of students. Other teachers have promised to call me before my next visit to 
discuss how I can best support their programs. As we get to know each other, we 
discover cooperative possibilities for change. 

The Consortium participants understandably, are significantly beyond my invol- 
vement in implementation. This will continue. Yet with the diversification of my 
participation, they will realize in themselves and with me, that formative evaluation 
is a source of program enrichment, a force to extend the implementation process. 
In this awareness, we elaborate our partnership in improvement. 



The three moments in formative evaluation, understanding, education, and action 
are interrelated aspects of an ongoing improvement process. Each moment has its 
emphasis; yet within each moment lies the other two. To live these moments requires 
a closeness among the formative evaluator and the program participants. At this 
point in the formative evaluat'»on of the Consortium, this closeness is uneven. 
There are signs of an emerging partnership, a sense of how we both are formative 
evaluators, as we sort out our connections in program improvement. I assume a 
major role in developing understanding while the Concortium participants do the same 
in those actions which facilitate implementation. We share equally in our mutual 
development. Our very closeness to the program and each other not only stimulates 
a partnership for improvement but also demands a fourth moment-reflexivity. 
Fourth Moment in Formative Evaluation: Reflexivity 
Closeness may become a problem for the formative evaluator if "going native" 
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is the result. The pejorative meaning of "going native" suggests that the formative 
evaluator identifies so much with the program participants that he or she loses pers- 
pective and the ability to be critical. Yet one mission of the formative evaluator 
is to get close enough to understand what it is like to«be a "native" in the program. 
The "native" is not necessarily mindless or one-sided about the program, but often 
capable of insightful reflection and judgment. To work like a "native" then, may 
provide the formative evaluator with a valuable sense of the program otherwise 
unattainable. Nonetheless, the formative evaluator is still confronted with the 
ongoing tensions in attempting to be both participant and observer, friend and 
stranger, insider and outsider, or developer and evaluator. To cope with these 
dilemmas, the formative evaluator relies on reflexivity (Siegle, 1986). - 

Reflexivity is about understanding and valuing subjectivity (Al lender, 1986; 
Reason, 1988), and its essential place in any human endeavour, including research 
and evaluation (HanunersTey S Atkinson, 1983). The formative evaluator is not a blank 
slate but brings a set of assumptions and beliefs to the task (Lincoln & Guba, 1985). 
This subjectivity likely will change throughout the history of formative evaluation 
and consequently, will become a part of what is studied and critiqued. In this 
"passionate scholarship" (Du Bois, 1983), the formative evaluator maintains a sen- 
sitivity to his or her role and its influence on the program, the participants, and 
the context. In short, reflexivity is a critical consciousness of self in relation 
to one's program and evaluation experiences. 

A critical self-consciousness is not restricted to the formative evaluator. 
Reflexivity compels both the formative evaluator and the program participants to 
pursue the fo*l lowing directions: 

Develop, implement, or evaluate programs whose significance is determined 

by personal choice and previous experience. 
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Examine their own intentions and actions while involved in program activities. 
Use self-inquiry as "a metaphor for elaborating the inquiry into the program. 
Alternate engagement and distantiation to penetrate how they influence and 
are influenced by the program and its evaluation. 

Maintain a "bifocular" critique, simultaneously interrogating the process 
of evaluating and probing the complexities of the program. 
Apply the same set of assumptions to frame their actions in and interpre- 
tations of the evaluation and the program. 

The formative evaluator and the program participants use their critical self- 
consciousness to enhance their joint involvement in the program. Reflexivity 
pervades the other three moments but in different forms — methodological reflexi- 
vity in understanding, interactive reflexivity in education, and programatic 
reflexivity in action. In the following sub-sections, I discuss the reflexive 
connection with the other three moments, noting particular principles and procedures 
and some of the issues which affect the connection in the formative evaluation of 
the Consortium. 

Understanding and Reflexivity 

Methodological reflexivity requires those who search for understanding to 
maintain a vigilant watch on" their own actions in inquiry. The "main instrument", 
under self-scrutiny is the formative evaluator, though .Tiethodological reflexivity 
also implicates program participants when they deliberately focus on sense-making 
activities. Critical self-consciousness in this form of reflexivity is a personal 
and introspective critique of how either the formative evaluator or the program 
participant builds understanding. It is a rigorous "subjectivity audit" (Krieger, 
1985; Peshkin, 1988) or "retrospective review" (Thiessen, 1988) intended to uncover 
how one's position in the program influences and is influenced by approacries to 
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understand, the program. 

The major concern in this introspective search for. connections is reciprocal 

reactivity. (Ruby, T982) calls this the "hall of mirrors": 

But in anthropology another layer may be entered into this equation: 
the effect of the anthropologist looking at the native looking at the 
anthropologist. We enter the hall of mirrors, the infinite regress, 
yet it is undeniably necessary. The subject changer* by being observed, 
and we must observe our impact on him or her and the resultant impact 
on ourselves and... (p. 19) 

Reciprocal reactivity sends the inquirer on parallel paths, critically examining 
the experiences of both participatingin and evatuating the programs. 

In the formative evaluation of the Consortium, I keep a journal of my perspec- 
tives and practices (Bogdan & Biklin, 1982; Burgess, 1984; Lincoln & Guba, 1985; 
Spradley, 1979). I describe my methodology, record comments I make to others about 
the evaluation process, and note what others say about my role. For example, some 
participants reacted: 

Oh, here comes the spy. 
Did you get that down? 

You take a lot of notes. What do you write about? 

You're just saying that to get us going. You' did that on 

purpose, you planned it all along." 
Where possible, I follow up such comments with a conversations about their 
views on my involvement. Their perceptions aid my self-analysis. On two occasions, 
I reviewed the journal, underlining key comments, transferring these comments on to 
file cards, and sorting these cards into categories. As a form of inductive analysis, 
it is similar to the interpretive procedures I use with the data from the Consortium 
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participants. Through the self-reviews, !• critique my actions, revise my next steps, 
and examine the implications of my emphasis for my understanding of and contribution 
to the Consortium. 

Methodological reflexivity for the Consortium participants, when evident, centres 
on their actions within the program. (See Action and Reflexivity below). It does not 
consider the processes they use to understand the Consortium's priorities or their 
role as the "main instruments" in their own formative evaluation. Those who use 
journals and videotapes tend to reflect on the instrumental effectiveness of their 
applications of cooperative learning and not on their position in the program. The 
Consortium coordinator and, to some extent the planning group members, demonstrate 
a collective type' of methodological reflexivity in their periodic program reviews. 
There are indications however, that methodological reflexivity develops more in 
association with the interactive reflexivity in education. 

Education and ReJ^eoci-utty 

Methodological reflexivity is a pause, a stepping back to be alone with oneself, 
to consider one's relationship with the program. It deepens the personal understanding 
of the improvement process. Interactive reflexivity is an ongoing phenomena, a 
social and empowering experience between the formative evaluator and the program 
participants. It broadens the basis for mutual development, the educational moment 
in formative evaluation, by raising the consciousness of eacli person as a teacher 
and learner in the program improvement process (Werner 1983) summarizes how under- 
standing, education, and reflexivity work together: 

Since critical reflection is a two-way street, it cannot be the task of an 
external 'expert' alone. Like a therapist, the evaluator is involved with people in 
mutual self-reflection. When engaged in questioning a school program he is also 
questioning himself. Initial critical reflection leads to further questions, which 
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in turn leads to a greater reflection. The process of reflection is dialectical, 
and the outcome of this dialectic is the mutual growth of self-understanding on 
the part of all who are involved. Therefore, the method and context is one of open 
dialogue and mutual questioning. All participants can question and be questioned. 
There are no positions of privilege, nor can critical sense-making be imposed upon 
someone. It must be open and free dialogue, (pp. 14-15) 

The sharing of critical self-consciousness between the formative evaluator and 
the program participants makes the exchange interactively reflexive (Rowan.^ 1981). 

Fundamentally, interactive reflexivity is about the ethics of empowerment. The 
formative evaluator and the program participants try to disrupt the conventionally 
separate and often unequal roles and become co-participants and co-evaluators. In 
the Consortium evaluation, we began this transformation in power with the following 
agreement: 

The participant an;d the evaluator will: 

negotiate the focus and the approaches of the evaluation. Though 
these will be determined at the beginning of each year, changes can 
be made at any time by mutual consent. 

respect the confidentiality of all conversations, documents, and 
(unless otherwise requested or permitted). 

help each other understand the information generated or shared in 
the evaluation. Through regular interaction, descriptions, inter- 
pretations, and ju.dgment will be checked and re-checked. Different 
positions or conclusions are acceptable and will be represented, 
decide which information will be reported to others Information 
will only be used in either oral or written communication after the 
participant and the evaluator agree on what Information will be released. 
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how it v/i''l be portrayed, and with whom it will be shared. 
Informed and mutual consent nave to occur in context and over time. A contract- 
like agreement neither ensures its practice nor equally empowers both parties. We 
often lapse into the old designations of the evaluator-as-expert and the participanfc- 
as-subjoct. Repeated discussions about their rights as participants and obligations 
as co-evaluator reminds the participants of the partnership we are attempting to 
create. 

When I ask the Consortium participants to discuss their experiences, it is a call 
for a reflexive account. Their comments however, often detach themselves from the 
program. For example, teachers talk about cooperative learning in the classroom 
without directly portraying their involvement in the process. We still learn about 
the program vicariously through tho exchange about each other's experiences. To 
enrich this further, I invite comments about myself and offer self-insights for their 
reaction. In this way, I encourage our interaction to become reflexive — "For in 
conversation, as in research, we meet ourselves" (Morgan, 1983, p. 404). If our present 
cooperation about program inquiry moves to a phase of interactive reflexivity, then 
the collaborative level of partnership is possible (Hord, 1986). 
Action and ^efl'exivity 

Programatic reflexivity necessitates all who facil ita'te implementation to evaluate 
their actions in relation to the priorities of the program. The main facilitators 
are the program participants themselves, though the formative evaluator is also 
reflexively implicated when he or she works like a participant and uses evaluation 
activities as an implementation strategy. Both the formative evaluator and the 
progi-am participants examine the extent to which their involvement is compatible 
with the intentions of the program, determine what can be done to enhance their 
involvement, and debate whether their involvement, the program priorities, or both 
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should be changed, Programatic reflexivity opens the implementation practices of 
each person to an indepth personal and social critique. 

One implication of programatic reflexivity is that I work within the principles 
of the Consortium, Lincoln and Guba (1985) embody this implication in their concept 
of value resonance. They state: 

The problem of value resonance is simply this: To the extent to which 
the inquirer's personal values, the axioms undergirding the guiding 
substantive theory, the axioms underlying the guiding methodological 
paradigm, and the values underlying the context are all consistent 
and reinforcing, inquiry can proceed meaningfully and will produce 
findings and interpretations that are agreeable from all perspectives. 
But, to the extent. to which they are dissonant, inquiry proceeds 'only with 
difficulty and produces findings and interpretations that are questionable 
and noncredible, (p,178) 
In principle, this is a comparatively easy challenge. The Consortium endorses inter- 
active professionalism, a notion which includes an ongoing exchange of resources, 
cooperative decision making, and shared expertise. These notions are consistent 
with the adaptive, deliberative, and integrative conditions of the qualitative paradigm. 
My research activities are resonant with the program. 

As I get close and act more like a Consortium participant however, I meet the 
personal conflict between authenticity and dissent, I must participate and be per- 
ceived to be participating authentically, genuinely facilitating irriplementation. It 
cannot be a peripheral act, a playing on the fringe, or a superficial performance, 
I m a colleague to the Consortium coordinator, a contributing member of the planning 
group, and a resource person. t<> some of the schools. In the spirit of the Consortium, 
I participate as a "native" to enrich the activities of the program. Yet is this 



23 



" 22 - 

involvement authentic especially when I disagree with some of the Consortium initia- 
tives? Can I facilitate implementation while at the same time increase my expressions 
of dissent? I have criticized the dominance of the one training model used to 
implement cooperative learning, suggested more inside-out change strategies (Hunt, 1987) 
and supported school-based implementation. These ideas are within the mandate of the 
Consortium but outside its present program. In my position as formative evaluator, I 
have the advantage of closeness unavailable to most of the program participants. Is 
my dissent constructive criticism or a subversive activity? I have posed this dilemma 
to the planning group for discussion and debate. 

Some of the planning group members and school-level Consortium participants 
experience a version of the same dilemma. One planning group member wonders if the 
translation of the priorities in her school board is a creative variation or a counter- 
force to their involvement in the Consortium. A school, deliberately uses cooperative 
learning as a vehicle to implement coaching and other collaborative working relationships 
instead of following the current Consortium emphasis of using coaching as a vehicle 
to implement cooperative learning. A few teachers have some reservations about the 
orientation to cooperative learning sponsored by the Consortium. In these instances. 
Consortium participants become self-conscious through authentic involvement in 
implementation and then criticize their involvement and the program. Programatic 
reflexivity enjoins us to live out the tensions between facilitating implementation 
and questioning the basis on which we support this implementation. 

Reflexivity is not only a check on unknowingly getting too close but a way of 
getting self-consciously close to the program. Methodological, interactive, and 
programatic reflexivity combine with the other moments — understanding, education. 
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and action — to create a more engaged and connected improvement process. At this 
point in the experiences of the Consortium, we are just beginning to penetrate this 
level of formative evaluation. 
Conclusion 

The 'Involvement with the Consortium provides a forum for considering the 
conceptual and empirical possibilities of formative evaluation. In the early stages, 
the designated formative evaluator and the program participants are in distinct 
positions withfewif any joint responsibilities. Their common commitment to program 
improvement initially leads to a range of experiences in three moments of formative 
evaluation ~ understanding, education, and action* As these moments increase in 
the development of the program, there is a movement towards and interaction between 
the designated formative evaluator and the program participants such that they both 
emerge together as collaborative formative evaluators with shared by differentiated 
expectations. The designated formative evaluator retains the central responsibility 
for building understanding while the program participants do the same for facilitating 
implementation. They assume equal responsibility in education, supporting the 
development of each other's program activities. Reflexivity, a fourth moment, adds 
a critical self-consciousness to the process transforming the relationship between 
the designated formative evaluator and the program participants into a genuine 
partnership. With the integration of the four moments, formative evaluation becomes 
a major force in program im[ movement. 
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